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Summary of Review 


This Center for American Progress report examines whether states’ adoption of stan- 
dards-based policies predicts low-income students’ NAEP achievement trends in fourth 
and eighth grade math and reading throughout the 2003-2013 decade. The report claims 
to analyze changes across five separate two-year intervals, but it only reports findings 
for 2009-2011, with no explanation of why or any documentation of the representative- 
ness of that single interval. The reported finding for the selected interval is that state 
adoption of standards policies positively predicts fourth (but not eighth) grade math 
NAEP and eighth (but not fourth) grade reading NAEP. Even these selected positive re- 
sults are statistically significant only at the generally unacceptable 0.10 level of signifi- 
cance. The report includes effect sizes but nothing about the percentage of the variance 
explained in their model. In short, the report does not adequately describe variables or 
analytic methods or completely report findings, and the data and methods used do not 
allow for any causal findings. They use state standards adoption across grade levels and 
subject areas as well as selected accountability mandates as predictors but fail to assess 
their quality or fidelity of implementation. Yet based on these very problematic and 
limited analyses, the authors conclude that their analysis “strongly supports the poten- 
tial of the Common Core to drive improvements in educational outcomes.” The study 
simply does not support this conclusion or the set of recommendations that follow. 
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REVIEW OF LESSONS FROM STATE PERFORMANCE 
ON NAEP: Wuy SoME HIGH-POVERTY STUDENTS 
SCORE BETTER THAN OTHERS 


Sharon L. Nichols, University of Texas at San Antonio 


I. Introduction 


The title of the Center for American Progress (CAP) report, “Lessons From State Perfor- 
mance on NAEP: Why Some High-Poverty Students Score Better than Others”! conveys a 
goal that is important and informative. After all, decades of educational policy efforts have 
focused on improving the educational outcomes of our poorest students.? The Elementary 
and Secondary Education Act of 1965 followed by No Child Left Behind Act of 2001 and 
more recently the Every Student Succeeds Act all have a core goal to improve educational 
conditions and outcomes for all of our students, especially our nation’s most vulnerable 
including minority, poor, ELL, and special education students. The title seems to promise 
some answers that are immediately telegraphed in the report’s table of contents: The dif- 
ference between high- and low-scoring states is the adoption of high standards. Specifically 
recommended is the Common Core. 


Standards-based policies have dominated the national landscape since the early 1990s. Al- 
though the conversation about standards dates back to the nineteenth century, the more 
recent impetus was sparked by the recommendations of the 1983 Nation at Risk report? that 
called for an overhaul to our nation’s educational system, advocating for “higher” standards 
and expectations for all of our students. Since that time, standards have been part of every 
Presidential administration’s education agenda. President Bill Clinton was an advocate for 
standards-based reform in Goals 2000, George W. Bush signed NCLB, formalizing the role 
of standards in education, and President Obama continued that legacy with his support 
of federally mandated, standards-based reform as mandated in NCLB and reinforced with 
School Improvement Grants and the Race to the Top initiative.5 The goal of CAP’s report is 
to examine the impact of these diverse policy efforts by asking whether increases in states’ 
adoption of standards-based reform policies under NCLB are related to low-income stu- 
dents’ achievement. 


Il. Findings and Conclusions of the Report 
The description of the analysis is incomplete and unclear rendering the findings confusing. 


The report claims that across the 2003-2013 decade, they “regressed the change in NAEP 
scores over a given two-year period onto policy changes from the same two-year period, con- 
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trolling for where they started.” This description implies they conducted 20 regression anal- 
yses to examine whether changes in states’ standards-based policies (the predictor variables) 
correlated with changes in low-income students’ NAEP achievement in fourth and eighth 
grade math and reading (the dependent variable) over five two-year time spans (2003-2005, 
2005-2007, 2007-2009, 2009-2011, 2011-2013). In the appendix, they “illustrate our mod- 
el” with results from only the 2009-2011 time period. The results from other time spans are 
not reported. Results from this one two-year period are interpreted causally as increases in 
standards-based reform policies cause increases (over the same time period) low-income 
students’ NAEP achievement in fourth math and eighth grade reading to increase -- but not 
fourth grade reading or eighth grade math. Even then the results are significant at only the 
generally unacceptable 0.10 level. Furthermore, the results suggesting an effect of only two- 
tenths of one NAEP point gain in fourth grade math and four-tenths of one NAEP point gain 
in eighth grade reading is basically meaningless in light of a 26-point Black-White achieve- 
ment gap® and a 24-point high and low income student achievement gap.’ 


It is conventional to report the multiple R when using regression approaches which tells 
the reader the magnitude of the relationships and the percent of the variance accounted 
for by their measure of policy adoption. This was not reported for any of the apparently 
20 analyses. They report using a “fixed-effects” regression model that would control for 
within state changes to some degree but that analysis is not presented nor explained. The 
authors report on four occasions that they supplemented their analysis with “anecdotal” 
or “impressionistic” interpretations of their data. On this foundation, the report concludes 
that standards-based reform is an effective policy strategy for increasing low-income stu- 
dent achievement. As a result, the report recommends that states should continue to em- 
brace standards-based reform and should widely adopt the Common Core State Standards 
to guide these efforts. 


Ill. The Report’s Rationale for Its Findings and Conclusions 


The report’s primary rationale is that if the scores they assigned to the “quality” of each 
state’s standards implementation have a positive relationship with test scores, then the ef- 
ficacy of standards-based reform is established. There are at least three problems with 
this rationale. First, as the authors admit, there is no measure of fidelity or intensity of 
implementation. Counting the quantity of standards adopted says little about the quality, 
extensiveness or fidelity of implementation. It is exceedingly difficult to objectively (and 
accurately) measure state-level policies and their implementation. Not only are policies in 
constant flux, but their implementation is widely uneven within and between states.® In- 
cluding an implementation fidelity measure is even more critical when results are used to 
make wide sweeping policy recommendations as is the case here. A second problem is that 
the authors contend a causal effect based on an unreported correlation. This is inappro- 
priate. Third, the report fails to account for numerous other variables that are known to 
influence student achievement such as student poverty, teacher turnover rates, and teacher 
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quality to name a few.? The absence of these considerations, the lack of valid predictors and 
the opaqueness of the analytic approach fundamentally invalidates the utility of the study’s 
rationale and conclusions. 


IV. The Report’s Use of Research Literature 


The research used to support the study relies heavily on a single study authored by Christo- 
pher Swanson and published by Education Week.*° Importantly, Swanson’s study is only one 
of a multitude of studies that have examined the connection between accountability-based 
policies and test results. This robust and growing literature base is notably absent from the 
report’s review." 


In fact, references to peer-reviewed empirical work that support or refute claims made 
throughout the report are sparse, outdated, or are descriptive rather than analytical. In the 
two pages dedicated to reviewing what we know about the effectiveness of standards-based 
reform, the report provides only 11 endnotes. Only one peer-reviewed empirical article was 
cited and this was a 2002” report. Three newspaper articles, three think tank reports (two 
of which are from the Center for American Progress), one book, and one blog round out the 
literature review. The citations are cherry-picked in that only two sentences are donated to 
one study that came to a contrary conclusion. This scant review ignores a rich literature that 
underscores much of what we know about the uneven standards-based policy implementa- 
tion throughout the NCLB era, as well as the inherent challenges and cautions associated 
with the literature’s mixed conclusions.¥ 


V. Review of the Report’s Methods 


As noted earlier, the methods suffer from (1) a lack of precision in describing key variables 
in the analysis, (2) the absence of key control variables in the analysis, and (3) selective re- 
porting of results. 


Standards-Based Policy Variable 


For every year in the study (2003-2013), a state was given a single policy “score” that al- 
legedly reflected the relative strength of each state’s standards-based policies. This score was 
based on three factors: the extent of standards adoption (subject matter and grade levels), 
the degree to which state standards were aligned with tests, and the number of test-based 
consequences mandated by state policy. A higher score meant “stronger” standards-based 
policy efforts. 
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These scores were the result of a combination of indicators compiled by the Education 
Counts Research Center (and sponsored by Education Week.) In the standards category, 
states could earn anywhere from 0-10 points across five different indicators, each of which 
have to do with whether standards are “clear, specific, and grounded in content.” States 
earned higher points if standards met this criterion in a greater number of subject areas 
and grade levels. In the assessment category, states could earn 0-6 points, earning more 
points for having tests aligned in more subject areas and for having tests that included short 
answer/extended response opportunities. Finally, the accountability category could earn a 
state from zero to four points with higher scores equating to a higher number of test-based 
sanctions (e.g., state sanctions low performing schools, requires exit exams, has report cards 
and/or rewards high-performing schools). Theoretically, then, a state could earn anywhere 
from 0-20 points based on how their policies were judged in any given year. 


Importantly, the varied distributions, relative importance, and weights of these three cat- 
egories mean that simply adding scores together is problematic. The report acknowledges 
this but used it anyway. The report fails to clearly explain their approach or rationale for 
combining the measures. All that is provided is that they “normed the results for each cat- 
egory to a final 10-point scale and averaged them.” There are various ways of weighting 
multiple measures into a single score (normalizing, factor analysis, professional judgment, 
etc.) but the reader is left adrift on this important point. 


In addition to not knowing how the policy predictor scores were computed, another perhaps 
even more vital problem has to do with meaningfulness of those scores. Assuming the trans- 
formation calculations were reliable, the report’s failure to account for fidelity of policy 
implementation creates a fundamental validity problem. Apparently, gains were ascribed 
and causal factors were confirmed based on eyeballing the data. 


Regression Analysis 


The report conducted regression analyses to see if changes in standards-based policy over 
time resulted in changes in low-income students’ NAEP performance in fourth and eighth 
grade math and reading. Using a regression model with a total n of only 50 is not recom- 
mended practice. The variables entered into the analyses are not well described nor even 
listed with weights in the conventional manner. The number of variables appears to be per- 
plexingly small in number consisting of three standards aggregate scores, an undefined state 
NAEP score from two years earlier, and a finance measure that weighed in at 0.0000. 


Somewhat puzzling was the use of a school funding equity measure (restricted range) that 
“expresses the difference between expenditures in the high- and lowest- spending districts” 
as a control.*® In such a model, the predictor and control variables should have some logical 
relationship to the question at hand, however, no rationale is provided for the inclusion of 
this measure as a control except its availability from Education Week. The restricted range 
is commonly used and measures finance inequities. The authors fail to include additional 
controls. 
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Lastly, the report says a fixed-effects regression model was used that would statistically 
control for within state variation. Although this logic is reasonable, there is no further de- 
scription or reporting of data describing or presenting such an approach. 


Missing Information 


As noted earlier, the regression results are not presented, the report provides only results 
from one set of analyses (2009-2011) out of a possible set of five, no correlation matrix is 
provided, no Multiple R is reported and variables and their weights are not provided. What 
makes this state of affairs even more troubling is that the report acknowledges the paucity 
of their findings but uses it as justification for substituting an ”anecdotal” and “impression- 
istic” analysis that is selective and incomplete. For example, in a review of the data patterns 
over time, the report notes that states such as “Kansas, Iowa, Idaho, Montana, and North and 
South Dakota showed some of the lowest gains on NAEP for low-income students since the 
late 1990s, and broadly speaking, these states have been less committed to standards-based 
reform.”’” These summary conclusions were not accompanied by data, making it impossible 
to evaluate these interpretations. The authors do note that confirmation bias may be at play. 


VI. Review of the Validity of the Findings and Conclusions 


There are significant and fundamental flaws in this report. The authors employ inappropri- 
ate research methods, fail to adequately define their approach, and do not report the find- 
ings from their analyses. Even if the limited data was taken at full value, the effects are trivi- 
al from a policy perspective. As a result, the authors fail to substantiate their conclusions. In 
short, the analysis does not provide evidence that “stronger” standards-based reform causes 
increases (or is even related to) low-income students’ achievement. The analysis is simply 
too incomplete to warrant such a finding. 


Their conclusion that the report justifies widespread adoption of the Common Core State 
Standards is not substantiated even on its own terms. The Common Core State Standards 
were not cycling through state adoption until 2010, are yet to be externally validated, have 
been adopted quite unevenly over time, and implementation has been haphazard.*® The re- 
port’s presentation that its results justify support for the CCSS in general is simply not sus- 
tained. 


VII. Usefulness of the Report for Guidance of Policy and Practice 


This report does not add to discussions of policy or practice. Even if the omissions and short- 
comings of this report were remedied, the analysis only provides a very narrow snapshot of 
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how policy might connect to practice. In this report, we only see one set of results from one 
point in time (and mostly before Common Core Standards were adopted, i.e., 2009-2011), 
but we are asked to rely on their unsubstantiated claim that standards-based policies work. 
The strident call for Common Core at the end of this report is misplaced given the mis- 
matched goals, questionable analysis and selected findings from the report. 
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